Skip to content

Add opt-in retry-with-backoff for rejected pushes - #1090

Open
monotek wants to merge 2 commits into
fluxcd:mainfrom
monotek:retry-push-conflict
Open

Add opt-in retry-with-backoff for rejected pushes#1090
monotek wants to merge 2 commits into
fluxcd:mainfrom
monotek:retry-push-conflict

Conversation

@monotek

@monotek monotek commented Aug 31, 2026

Copy link
Copy Markdown

Why

When many independent ImageUpdateAutomation objects push to the same branch of one GitOps repo (e.g. one IUA per service/region, all on independent 5-minute reconcile timers), a multi-service release can cause many of them to try pushing around the same time. A concrete example observed in production: five regions all resolved a new image tag within 1-2 minutes of each other, but the resulting git commits landed 13-29 minutes apart, because several IUAs lost a non-fast-forward push race against others pushing to the same branch:

14:32:43  Warning  failed to update source: failed to push to remote: object not found
14:45:40  Normal   pushed commit 'a622711' — Release java-backend:2026.34.2

There is no in-process retry today — the next attempt only happens on the next scheduled reconcile, or via controller-runtime's per-item exponential-backoff requeue (750ms doubling, capped at 15min). Under heavy contention, a single IUA can lose several races in a row, each costing a full backoff cycle, which is how a sub-minute git operation turns into a 29-minute delay. This reproduces on every release with enough concurrent writers to one branch, and gets worse as more services onboard per-instance IUAs.

A retry-with-backoff loop that reuses the existing local clone (fetch + reset instead of a full re-clone) resolves a lost race in seconds instead of minutes, following the same shape as a standard retry_with_backoff(5, 2, ...) wrapper around a git pull --rebase && git push cycle.

What

  • commitAndPushWithRetry (new internal/controller/push_retry.go) wraps CommitAndPush: on a push rejected specifically because another writer already advanced the branch (source.IsPushConflict), it waits (2s/4s/8s/16s backoff, 5 attempts total), fetches and hard-resets onto the new remote tip (SourceManager.RefreshToRemote, no full re-clone), re-applies policies against the refreshed tree, and retries the commit and push. Any other error, or exhaustion of all attempts, is returned unchanged — existing condition/error handling in reconcile() is untouched.
  • The retry loop is bounded by min(.spec.interval / 2, 2 minutes) so contention on one branch can't block a reconcile worker indefinitely and starve other ImageUpdateAutomation objects sharing the pool.
  • Depends on (*gogit.Client).FetchAndReset and git.ErrPushRejected from git/gogit: add FetchAndReset and ErrPushRejected for push-conflict recovery pkg#1289 (companion PR) — go-git has no rebase API, so this fetches + hard-resets instead of a real rebase, which reaches the same end state here since policies are re-applied fresh against the reset tree rather than blindly replaying a stale diff.
  • New feature gate GitPushRetryOnConflict, disabled by default — opt-in, so this ships as an explicit choice rather than a behavior change for existing installs.

Testing

  • internal/source: TestSourceManager_RefreshToRemote_RecoversFromPushConflict — forces a real non-fast-forward rejection via a competing out-of-band push, then proves recovery lands the new commit on top of the competing one (parent-hash check), not an overwrite.
  • internal/controller: five tests covering recovery within N attempts, exhausting all attempts against a real conflict, a non-conflict error (connection failure) returning after exactly one attempt, retry-budget enforcement under a short .spec.interval, and the feature gate's off-by-default fallback path.
  • Full go test ./... passes (envtest via make install-envtest), go vet/gofmt clean.

Notes for reviewers

When many independent ImageUpdateAutomation objects push to the same
branch of one GitOps repo, a rejected push (another writer already
advanced the branch) previously just failed the reconciliation, with the
next attempt only happening on the next scheduled reconcile or via
controller-runtime's exponential-backoff requeue. Under heavy write
contention this turns a sub-minute git operation into a delay of tens of
minutes.

Add commitAndPushWithRetry, which on a push rejected specifically due to
a conflict (source.IsPushConflict) fetches and hard-resets onto the new
remote tip (SourceManager.RefreshToRemote, no full re-clone), re-applies
policies against the refreshed tree, and retries the commit and push, up
to 5 attempts with exponential backoff (2s/4s/8s/16s). Any other error,
or exhaustion of all attempts, is returned unchanged. The retry loop is
bounded by min(.spec.interval / 2, 2 minutes) so contention on one branch
cannot block a reconcile worker indefinitely.

Gated behind the new GitPushRetryOnConflict feature gate, disabled by
default.

Depends on FetchAndReset and ErrPushRejected from a companion fluxcd/pkg
change; go.mod currently points at that change's fork branch pending a
tagged release.

Assisted-by: Claude Sonnet 5/claude-sonnet-5
Signed-off-by: André Bauer <monotek23@gmail.com>
Signed-off-by: André Bauer <monotek@users.noreply.github.com>
@monotek
monotek marked this pull request as ready for review September 1, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant